Figure 4: Evaluation of model performance

This notebook generates individual panels of Figure 4 in "Combining satellite imagery and machine learning to predict poverty".


In [153]:
from fig_utils import *
import matplotlib.pyplot as plt
import time

%matplotlib inline

Transfer learning vs. nightlights

In these experiments, we compare the performance of the transfer learning model based on satellite imagery with the performance of a model that uses nightlights.

The parameters needed to produce the plots for Panels A and B are as follows:

  • country_paths: Paths of directories containing pooled survey data
  • percentiles: Wealth percentiles to evaluate
  • survey: Either 'lsms' or 'dhs'
  • dimension: Number of dimensions to reduce image features to using PCA
  • k: Number of cross validation folds
  • k_inner: Number of inner cross validation folds for selection of regularization parameter
  • trials: Number of trials to average over
  • poverty_line: International poverty line ($1.90/capita/day)
  • multiples: Multiples of the poverty line to plot

For many trials, it will take several minutes or more to produce the plots. For 100 trials, it should take 40-60 minutes for LSMS and longer for DHS.

Each data directory should contain the following 4 files:

  • conv_features.npy: (n, 4096) array containing image features corresponding to n clusters
  • nightlights.npy: (n,) vector containing the average nightlights value for each cluster
  • households.npy: (n,) vector containing the number of households for each cluster
  • image_counts.npy: (n,) vector containing the number of images available for each cluster

Each data directory should also contain one of the following:

  • consumptions.npy: (n,) vector containing average cluster consumption expenditures for LSMS surveys
  • assets.npy: (n,) vector containing average cluster asset index for DHS surveys

Exact results may differ slightly with each run due to randomly splitting data into training and test sets.

Panel A: Pooled LSMS


In [148]:
country_path = '../data/output/LSMS/pooled/'
percentiles = [0.05, 0.10, 0.15, 0.20, 0.25, 0.30, 0.35,
               0.40, 0.45, 0.50, 0.55, 0.60, 0.65, 0.70,
               0.75, 0.80, 0.85, 0.90, 0.95, 1.00]
survey = 'lsms'
dimension = 10
k = 10
k_inner = 5
trials = 5
poverty_line = 1.90
multiples = [1, 2, 3]

In [149]:
t0 = time.time()
compare_models(country_path, survey, percentiles, dimension, k, k_inner,
             trials, poverty_line, multiples)
t1 = time.time()
print 'Finished in {} seconds'.format(t1-t0)


Finished in 121.461523056 seconds

Panel B: Pooled DHS


In [151]:
country_path = '../data/output/DHS/pooled/'
percentiles = [0.05, 0.10, 0.15, 0.20, 0.25, 0.30, 0.35,
               0.40, 0.45, 0.50, 0.55, 0.60, 0.65, 0.70,
               0.75, 0.80, 0.85, 0.90, 0.95, 1.00]
survey = 'dhs'
dimension = 10
k = 5
k_inner = 3
trials = 3

In [152]:
t0 = time.time()
compare_models(country_path, survey, percentiles, dimension, k, k_inner,
             trials, poverty_line, multiples)
t1 = time.time()
print 'Finished in {} seconds'.format(t1-t0)


Finished in 187.415472984 seconds

Randomization tests

In these experiments, we randomly reassign daytime imagery to survey locations and retrain the model on incorrect images (see SM 1.7).

The parameters needed to produce the plots for Panels C and D are as follows:

  • country_names: Names of countries as a list of lower-case strings
  • country_paths: Paths of directories containing survey data corresponding to specified countries
  • survey: Either 'lsms' or 'dhs'
  • dimension: Number of dimensions to reduce image features to using PCA
  • k: Number of cross validation folds
  • k_inner: Number of inner cross validation folds for selection of regularization parameter
  • points: Number of regularization parameters to try
  • alpha_low: Log of smallest regularization parameter to try
  • alpha_high: Log of largest regularization parameter to try
  • trials: Number of trials for shuffled distribution

If trials is large (>100), producing the plots will take more than a couple of minutes.

Each data directory should contain the following 3 files:

  • cluster_conv_features.npy: (n, 4096) array containing image features corresponding to n clusters
  • cluster_households.npy: (n,) vector containing the number of households for each cluster
  • cluster_image_counts.npy: (n,) vector containing the number of images available for each cluster

Each data directory should also contain one of the following:

  • cluster_consumptions.npy: (n,) vector containing average cluster consumption expenditures for LSMS surveys
  • cluster_assets.npy: (n,) vector containing average cluster asset index for DHS surveys

Exact results may differ slightly with each run due to randomly splitting data into training and test sets.

Panel C: LSMS consumption expenditures


In [50]:
# Parameters
country_names = ['nigeria', 'tanzania', 'uganda', 'malawi', 'pooled']
country_paths = ['../data/output/LSMS/nigeria/',
                '../data/output/LSMS/tanzania/',
                '../data/output/LSMS/uganda/',
                '../data/output/LSMS/malawi/',
                '../data/output/LSMS/pooled/']
survey = 'lsms'
dimension = 100
k = 3
k_inner = 3
points = 10
alpha_low = 0
alpha_high = 3
trials = 100

In [51]:
run_randomization_test(country_names, country_paths, survey, dimension,
                       k, k_inner, points, alpha_low, alpha_high, trials)


Panel D: DHS assets


In [52]:
# Parameters
country_names = ['nigeria', 'tanzania', 'uganda', 'malawi', 'rwanda', 'pooled']
country_paths = ['../data/output/DHS/nigeria/',
                '../data/output/DHS/tanzania/',
                '../data/output/DHS/uganda/',
                '../data/output/DHS/malawi/',
                '../data/output/DHS/rwanda/',
                '../data/output/DHS/pooled/']
survey = 'dhs'
dimension = 100
k = 3
k_inner = 3
points = 10
alpha_low = 0
alpha_high = 3
trials = 100

In [53]:
run_randomization_test(country_names, country_paths, survey, dimension,
                       k, k_inner, points, alpha_low, alpha_high, trials)



In [ ]: